Systems and methods for multi-user shared virtual and augmented reality-based haptics

ABSTRACT

Systems and methods for multi-user shared virtual and augmented reality-based haptics are disclosed. One illustrative method for multi-user shared virtual and augmented reality-based haptics includes determining a position of an object; determining a viewpoint of at least one observer with respect to the object; determining a haptic effect to be output based at least in part on the position and the viewpoint; and outputting the haptic effect.

FIELD OF THE INVENTION

The present application relates to the field of virtual and augmentedreality applications and devices. More specifically, the presentapplication relates to multi-user shared virtual and augmentedreality-based haptics.

BACKGROUND

Virtual and augmented reality systems are becoming more prevalent. Suchsystem allow a user to interact with virtual objects within a virtualenvironment. Some such systems allow a third party to view what the useris experiencing in the virtual environment. But interaction between usescan be limited. Systems and methods for allowing a shared experience ina virtual or augmented reality environment are needed.

SUMMARY

In one embodiment, a method for multi-user shared virtual and augmentedreality-based haptics comprises determining a position of an object;determining a viewpoint of at least one observer with respect to theobject; determining a haptic effect to be output based at least in parton the position and the viewpoint; and outputting the haptic effect. Inanother embodiment, a computer readable medium may comprise programcode, which when executed by a processor is configured to enable theabove described method.

These illustrative embodiments are mentioned not to limit or define thelimits of the present subject matter, but to provide examples to aidunderstanding thereof Illustrative embodiments are discussed in theDetailed Description, and further description is provided there.Advantages offered by various embodiments may be further understood byexamining this specification and/or by practicing one or moreembodiments of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure is set forth more particularly in theremainder of the specification. The specification makes reference to thefollowing appended figures.

FIG. 1 shows an illustrative system for multi-user shared virtual andaugmented reality-based haptics in one embodiment.

FIG. 2 shows another illustrative exemplary virtual or augmented realitysystem in one embodiment.

FIG. 3 is depicts an artist's rendering within a virtual or augmentedreality environment in one embodiment.

FIGS. 4 through 15 illustrate a process for mixing and applying colorsto an environment in one embodiment.

FIG. 16 is an illustration of a user interface for varying a tool in oneembodiment.

FIG. 17 illustrates a palette for creating paths, such as roads, in oneembodiment.

FIGS. 18-24 illustrate various states that may be utilized in someembodiments.

FIGS. 25-32 illustrate various embodiments for interacting with avirtual environment with a “magic window” application.

FIGS. 33 and 34 illustrate embodiments of a content-creator applicationuser interface.

FIG. 35 illustrates a tablet that can be used in conjunction withvarious embodiments.

FIGS. 36-38 illustrate an embodiment in which the orientation in whichthe user holds the tablet controls the functionality available to theuser.

FIG. 39 is an illustration of how a tablet may be laid out for use in avirtual or augmented reality environment in one embodiment.

FIG. 40 is an illustration of a primitive object used to generate morecomplex objects in a virtual or augmented environment in one embodiment.

FIG. 41 is a flowchart illustrating a method according to oneembodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternativeillustrative embodiments and to the accompanying drawings. Each exampleis provided by way of explanation, and not as a limitation. It will beapparent to those skilled in the art that modifications and variationscan be made. For instance, features illustrated or described as part ofone embodiment may be used in another embodiment to yield a stillfurther embodiment. Thus, it is intended that this disclosure includemodifications and variations as come within the scope of the appendedclaims and their equivalents.

Illustrative Example System for Multi-User Shared Virtual and AugmentedReality-Based Haptics

In one illustrative embodiment, a mobile application executing on adigital pad or other mobile device enables a user to connect to andinteract with virtual objects in a virtual or augmented realityenvironment. The user may also create objects, or the objects may becreated via different users, including, for example artists or othercontent creators. Any of the users may create, observe, or interact withthe objects using a mobile pad, a head-mounted display (“HMD”), or anyother device. Both the HMD and mobile devices may include embeddedhaptic actuators. For example, in one embodiment, an artist in anAugmented Reality (AR) environment interacts with a 3D world, creatingvirtual objects and possibly interacting with physical objects. Thegeneral idea is to enable observers who are not wearing the headset toparticipate in the AR experience either through observing (“magicwindow”) or collaborating.

The user of the pad can experience haptics based on the user'sperspective relative to the virtual or augmented environment. Forexample, the user of the pad may experience the environment from theperspective of an observer or through the perspective of an avatarinteracting with the environment. The user of the pad experiences hapticeffects based on the environment, which may vary, the user'sperspective, which may also vary, and interactions between the user andthe environment. The user may be one of a number of users allexperiencing and interacting with the same virtual or augmented realityenvironment at once. And each user may have her own perspective or mayshare a perspective with another user.

The haptic effects may be based on an object that is treated as thehaptic effect emitter. The emitter may be directional and may have astrength that diminishes as a user moves further away from the emitterin the virtual or augmented environment. Further the effect may bemultidirectional based on the user's position with respect to theemitter. The emitter could be an object or a location.

Such a system allows multiple users to experience a common, virtualenvironment. And they can do so either in a collocated or non-collocatedreal world space. Each user may experience the environment using adifferent type of device. In some embodiments, a haptic designer can usesuch a system to create and transmit haptic effects to multiple users.And the effects can vary based on the relative position of the userswithin the virtual or augmented environment.

In one such system, users can experience the environment form the pointof view of a content creator, either by creating content themselves orby experiencing the environment through the point of view of anotheruser. The user can also experience the environment by participating init. For example, an artist may create a road or a flight path. Then theuser may experience the road from the point of view of a car travelingthe road. In another example, a product designer might create a virtualproduct for other users to experience. The perspective and theenvironment both affect the types of experience, including the hapticeffects, that are provided to the user.

In an augmented reality embodiment, the environment may be grounded to atracking base or other grounding element, such as a QR code. And theexperience may be collocated or, alternatively, be cloud based such thatusers need not be near one another to share the experience.

The preceding example is merely illustrative and not meant to limit theclaimed invention in any way.

Illustrative Systems for Multi-User Shared Virtual and AugmentedReality-Based Haptics

FIG. 1 shows an illustrative system 100 for multi-user shared virtualand augmented reality-based haptics. The device shown in FIG. 1 may beused by a user experiencing a shared environment with an artist or bythe artist. Particularly, in this example, system 100 comprises acomputing device 101 having a processor 102 interfaced with otherhardware via bus 106. A memory 104, which can comprise any suitabletangible (and non-transitory) computer-readable medium such as RAM, ROM,EEPROM, or the like, embodies program components that configureoperation of the computing device. In this example, computing device 101further includes one or more network interface devices 110, input/output(I/O) interface components 112, and additional storage 114.

Network device 110 can represent one or more of any components thatfacilitate a network connection. Examples include, but are not limitedto, wired interfaces such as Ethernet, USB, IEEE 1394, and/or wirelessinterfaces such as IEEE 802.11, Bluetooth, or radio interfaces foraccessing cellular telephone networks (e.g., transceiver/antenna foraccessing a CDMA, GSM, UMTS, or other mobile communications network(s)).

I/O components 112 may be used to facilitate connection to devices suchas one or more displays, touch screen displays, keyboards, mice,speakers, microphones, cameras, and/or other hardware used to input dataor output data. Storage 114 represents nonvolatile storage such asmagnetic, optical, or other storage media included in device 101.

System 100 further includes a touch surface 116, which, in this example,is integrated into device 101. Touch surface 116 represents any surfacethat is configured to sense touch input of a user. One or more sensors108 are configured to detect a touch in a touch area when an objectcontacts a touch surface and provide appropriate data for use byprocessor 102. Any suitable number, type, or arrangement of sensors canbe used. For example, resistive and/or capacitive sensors may beembedded in touch surface 116 and used to determine the location of atouch and other information, such as pressure. As another example,optical sensors with a view of the touch surface may be used todetermine the touch position.

In some embodiments, sensor 108, touch surface 116, and I/O components112 may be integrated into a single component such as a touch screendisplay. For example, in some embodiments, touch surface 116 and sensor108 may comprise a touch screen mounted overtop of a display configuredto receive a display signal and output an image to the user. The usermay then use the display to both view the movie or other video andinteract with the haptic generation design application.

In other embodiments, the sensor 108 may comprise an LED detector. Forexample, in one embodiment, touch surface 116 may comprise an LED fingerdetector mounted on the side of a display. In some embodiments, theprocessor 102 is in communication with a single sensor 108, in otherembodiments, the processor 102 is in communication with a plurality ofsensors 108, for example, a first touch screen and a second touchscreen. The sensor 108 is configured to detect user interaction and,based on the user interaction, transmit signals to processor 102. Insome embodiments, sensor 108 may be configured to detect multipleaspects of the user interaction. For example, sensor 108 may detect thespeed and pressure of a user interaction and incorporate thisinformation into the interface signal.

Device 101 further comprises a haptic output device 118. In the exampleshown in FIG. 1A haptic output device 118 is in communication withprocessor 102 and is coupled to touch surface 116. The embodiment shownin FIG. 1A comprises a single haptic output device 118. In otherembodiments, computing device 101 may comprise a plurality of hapticoutput devices. The haptic output device may allow a user or artist toexperience effects as they are generated in relation to the virtual oraugmented reality environment or create haptic effects to beincorporated into the overall experience.

Although a single haptic output device 118 is shown here, embodimentsmay use multiple haptic output devices of the same or different type tooutput haptic effects. For example, haptic output device 118 maycomprise one or more of, for example, a piezoelectric actuator, anelectric motor, an electro-magnetic actuator, a voice coil, a shapememory alloy, an electro-active polymer, a solenoid, an eccentricrotating mass motor (ERM), or a linear resonant actuator (LRA), a lowprofile haptic actuator, a haptic tape, or a haptic output deviceconfigured to output an electrostatic effect, such as an ElectrostaticFriction (ESF) actuator. In some embodiments, haptic output device 118may comprise a plurality of actuators, for example a low profile hapticactuator, a piezoelectric actuator, and an LRA. Additional actuators maybe incorporated into devices in communication with computing device 101.For example, a head-mounted display may incorporate actuators that allowit to provide haptic effects to an artist or any other user interactingwith the environment.

Turning to memory 104, exemplary program components 124, 126, and 128are depicted to illustrate how a device may be configured to determineand output haptic effects. In this example, a detection module 124configures processor 102 to monitor touch surface 116 via sensor 108 todetermine a position of a touch. For example, module 124 may samplesensor 108 in order to track the presence or absence of a touch and, ifa touch is present, to track one or more of the location, path,velocity, acceleration, pressure, and/or other characteristics of thetouch over time.

Haptic effect determination module 126 represents a program componentthat analyzes data regarding audio and video characteristics to select ahaptic effect to generate. Particularly, module 126 comprises code thatdetermines, based on the audio or video properties, an effect togenerate and output by the haptic output device. Module 126 may furthercomprise code that selects one or more existing haptic effects toprovide in order to assign to a particular combination of properties,such as perspective, distance, and properties of the environment. Forexample, a user experiencing an environment as a participant ridingalong a gravel road, may experience a coarse, relatively high magnitudehaptic effect. Different haptic effects may be selected based on variouscombination of these features. The haptic effects may be provided viatouch surface 116 and may be provided to an artist or other hapticdesigner in order that the designer can preview a particular effect orset of effects and modify them as necessary to better model the scene.

Haptic effect generation module 128 represents programming that causesprocessor 102 to generate and transmit a haptic signal to haptic outputdevice 118, which causes haptic output device 118 to generate theselected haptic effect. For example, generation module 128 may accessstored waveforms or commands to send to haptic output device 118. Asanother example, haptic effect generation module 128 may receive adesired type of haptic effect and utilize signal processing algorithmsto generate an appropriate signal to send to haptic output device 118.As a further example, a desired haptic effect may be indicated alongwith target coordinates for the texture and an appropriate waveform sentto one or more actuators to generate appropriate displacement of thesurface (and/or other device components) to provide the haptic effect.Some embodiments may utilize multiple haptic output devices in concertto simulate a feature. For instance, a variation in texture may be usedto simulate crossing a boundary between buttons on an interface while avibrotactile effect simulates the response when the button is pressed.

FIG. 2 is a diagram illustrating an exemplary virtual or augmentedreality system 200. In the environment shown, and HMD 202 allows a userto experience the environment in an immersive manner. The HMD 202includes displays that allow a user to see the environment. The HMD 202may also include speakers, such as earphones that allow a user to hearsounds associated with the environment. The HMD 202 may also includehaptic actuators so that a user can feel sensations associated with theenvironment.

The system 200 also includes one or more detectors 204 a, 204 b. Thedetectors 204 sense movement of the HMD 202 in the environment andprovide that information to a processor (not shown) located in, near, orremote from the system 200. The environment may include other devices.For example, the user may be wearing or holding various devices thatallow the system to detect movement of various parts of the user's body.For example, the user may pick up a controller to use as a paint brush.The sensors 204 can detect the user's gestures with the device andprovide signals to the processor reflecting the movement. Any type ofmotion tracking system could be utilized including, for example,infrared, electromagnetic, RGB Camera-based, or others.

Examples of Multi-User Shared Virtual and Augmented Reality-BasedHaptics

Embodiments of multi-user shared virtual and augmented reality-basedhaptic systems may receive a variety of inputs. For example, one or moreusers, such as an artist, observer, content creator, or designer mayinteract with the VR or AR environment. In one such embodiment, sensordata indicating, for example hand position or interaction with a virtualpalette or VR or AR objects may be used to determine how the environmentand the perspective of the user is changing. Such information mayinclude, for example, proximity to virtual or physical object thatserves as a haptic emitter. Other information that reflects the spatialrelationship between a user and emitter may be utilized as well.

In one embodiment, in which a user acts as an artist, a system receivesdata regarding various brushes and brushstrokes that available for theartist to use to create or augments objects in the environment. The datamay include information about changes to the size of a brushstroke orabout the “paint” that is used within the environment. The brushstrokesmight create simple visual effects. Alternatively, the brushstrokes maybe used to create virtual objects with which the artist or another user,such as a third party observer, can interact. For instance, an artistmight draw or paint a virtual road for an automobile or a virtual flightpath for a plane, insect, or bird. The artist or user might then add anautomobile or airplane to the environment. The vehicle can then followthe road or flightpath through the virtual environment. The users' andthe vehicle's interaction with the environment may result in thecreation and outputting of various haptic effects.

Various other embodiments and features are possible. For example, theartist may render a boat in water or a horse on a trail. The artistcould also create hiking trails that can be traversed or viewed.Properties of the various features can then be used to vary hapticeffects output to a user of the system. For example, the trail or roadcould be built using asphalt, sand, or gravel. Haptics output to a userof the system could vary based on the type of material used to build theroad. And the environment could be changed by any of the uses, resultingin a collaboratively-built environment.

Such an environment could be crated in any number of ways. A user,including a content creator, can position themselves within theenvironment and vary their position. Further, the objects that the usercreates can be moved and modified. For example, a first user couldcreate a road. Then a second user could create or spawn a car totraverse the road and thereby change the environment. A third user couldthen control the car, starting, stopping, and varying the speed of thecar. Such objects could exist virtually or in combination with a realobject or in some combination of both. For example, a real object mightbe augmented with a component overlaid on the real object. The augmentedcomponent can be any combination of modalities, such as a visual objectwith haptics or could be haptic effects alone. In other words, aphysical object could have haptics overlaid upon it even if the visualaspects of the object were not changed.

The experience of a user could vary based on their position relative toother users or to objects in the environment capable of emitting hapticeffects. For example, the distance between the user and an object in theenvironment could affect the haptic effects the user experienced. Therelative position of the user and the object may be variable since theuser and the virtual object may both be moving and thus may be movingrelative to one another. In one embodiment, the haptic effectexperienced by the user may depend on the velocity with which the twouser and an object are approaching or departing from one another.

The user may interact with objects, virtual, real, or some combination,or with the environment itself. For instance, the user may add an objectto the environment and then interact directly or indirectly with thatobject. For example, if the user adds (spawns) a car on a road drawn bythat user or another user. The user can then provide input to controlthe car, including, for example starting, stopping, and changing speed.All of these interactions can cause haptic effects to be output or tochange. In another example, the user may cause a palette to bedisplayed, and use the palette to add visual elements to theenvironment. The user could also change the environment more generally,such as by changing the lighting (day or night), or adding some othersort of environmental effect to the environment (clouds, wind, rain,etc.). Observers can affect the environment in various different ways asthey interact with it.

In some embodiments, the viewpoint of a user with respect to objectswithin the environment can affect the user's experience, such as byvarying the haptic effects output to the user. If multiple users aresimultaneously experiencing the environment, then each user mayexperience a different set of effects. For instance, a user mayexperience the environment from a first person perspective. Forinstance, a first user and a second user may experience the environmentbased on their location relative to one another, such as when a thirdparty observer views an artist rendering objects in a virtualenvironment. This perspective can change as the two users move relativeto one another. In another example, a product designer could create aproduct in the environment, while another user moves around the productexperiencing various aspects of the product and potentially makingchanges to the product as it is designed. All of these interactions maytrigger distinct haptic effects.

In some embodiments, a user may experience the environment as athird-party observer or from the perspective another user, such as anartist creating content in the environment. In other embodiments, theuser may experience the environment from the perspective of aparticipant in the environment, such as for example, sitting in a cartraversing a road or in an airplane traversing a flight path.

The viewpoint of such a user may vary as the user, the person from whoseviewpoint the user is experiencing the environment, or the object moveswithin the environment. In some environments, the user is not “inside”the environment. For instance, a content creator creates a virtual carwith haptics as an advertisement. A user could experience the car in ARor VR using an HMD, or alternatively, they could also see atwo-dimensional view of the car on a website, where the proximity of thecursor to the haptic emitter (the car) comprises the virtual spatialproximity for creation of haptic effects.

The user's experience may also be affected by the user's viewpoint withrespect to the object from which the haptic effect is emitted. Forexample, the haptic effect could result from a direct interaction withan object. For instance, an artist could mix paint within theenvironment. The artist's interaction with the paint might trigger ahaptic effect. In another example, a user interacts as a third-partyobserver with respect to where the haptics are being emitted. Forinstance, a bullet approaching the user might trigger a particularhaptic effect. As another example, wind occurring in the environmentwould have different effect depending upon where the user is located inrelation to the wind and would cause the user to experience differenthaptic effects. In yet another example, a user might pass through atunnel in the environment, which would affect the effects provided tothe user.

In some embodiments of the invention, the user may not see visualobjects in the environment but still be able to experience other aspectsof the environment, such as the haptic effects. For example, a user maybe “inside” environment but unable to see one or more objects present inthe environment. For instance, the environment could be dark. As theuser interacts with the environment, such as by touching or waling overobjects, haptics can be provided to the user. In some embodiments, theposition of various users with respect to one another, either in the VRor AR environment or in the physical world, can affect the hapticeffects provided to the user.

Embodiments of the invention can provide a variety of haptic effects.The haptic effects may be determined and generated based on any of thevariables above, such as position or viewpoint. The haptic effect orparameters of a haptic effect may be calculated or synthesized using aphysics models. For instance, a haptic effect may be calculated based onan incline that a vehicle is traversing or a barrier through which avehicle is passing.

A variety of types of haptic effects may be utilized, including forexample friction, vibration, and air, and kinesthetic force. The hapticscan be used to indicate contact, air movement, water, and otherconditions or objects present in the virtual or augmented environment.Examples of such haptic effects include a haptic effect based on atexture of the road created by the artist. Another example is a hapticeffect based on a velocity of a car as the car drives around a road. Thehaptic effect may be based on a user's interaction with the environmentgenerally, such as by moving through the environment without touching aspecific object. In some embodiments, the haptic effect may bedirectional and increase or decrease in proportion with distance fromthe emitter. For example, the haptic effect may push or pull the user.

In some embodiments, various properties of a haptic effect may vary,including frequency or pitch, magnitude, amplitude, directionality of ahaptic source, or directionality of actuator. In some embodiments,multiple effects may be combined. Also, a haptic effect may vary basedon the type of device used to output the haptic effect. As one example,a user may add an oil slick or spot to a rough road within theenvironment and thereby cause a haptic effect associated with the roadto change from a haptic effect associated with a rough texture to oneassociated with a smooth texture, e.g., a course, strong effect maybecome more finely grained and weaker and thus signal that the road isslippery.

In another example embodiment, an artist is using a particular vibrantcolor to create an object in an environment and a strong haptic effectis output when the artist or another user interacts with the object. Theartist can then change a property of the object (e.g., change the colorto something less vibrant), and the haptic effect associated with thevirtual object changes, becoming less strong. In other embodiments, thehaptic effect changes depending on the type of tool the user uses tocreate or modify objects in the environment.

In some embodiments, the haptic effects change as the viewpoint of theuser changes. For example, stronger haptics or additional haptics may beoutput when the user switches from a first person mode to a third personmode. In another embodiments, the haptics may change based on position.For example, if the user is in the center of the environment, theeffects can be scaled based on proximity to this position. In anotherexample, a user's viewpoint is from the center of a road, and so hapticsassociated with cars driving around the road can be increased to createsensation of being positioned at the center (e.g., a center of a racetrack).

In some embodiments, the haptic effects vary based on the user movingtheir head to look around a three hundred and sixty degree environment.For example, a first haptic effect may be outline when a user is lookingat an object in front of the user, and a second haptic effect can beoutput when the user turns around and looks at an object behind theuser. Haptic effects may also be panned (e.g., from left to right) asthe point of view of the user changes. In yet further embodiments,haptic effects vary based at least in part on changes in the physicallocation of the user within or around the environment.

For example, if the 3rd party observer and artist are collocated, as the3rd party observer moves closer to, or away from, the artist, hapticeffects can become stronger (or weaker).

Haptic effects can be changed based on the position of 3rd partyobserver relative to a virtual object. For example, in one embodiment, afirst user creates a fire effect using a fire brushstroke, and a hapticeffect associated with the fire brushstroke (e.g., a high frequency orhigh pitched haptic effect to indicate heat or fire, or a thermal hapticeffect) is output. As a second user moves away from the fire, thefrequency of the haptic effect can change indicating that the seconduser is becoming cooler, or the haptic temperature can become lower.

Haptic effects can also be changed based on the viewpoint of the hapticeffect (e.g., position of the source or emitter of the haptic effect)and/or position of a user relative to the source of the haptic effect.For example, a pitch of a haptic effect associated with a second carapproaching a first car in which a user is “riding” can change as thetwo cars move toward (or away from) each other. Another user mayexperience the same approach of the two cars from an observer viewpointand experience a Doppler effect in the haptic effects as the carsapproach or move away from the observer's position.

Depending on the capabilities of the system, certain haptic effects maybe prioritized over other effects. For instance, in some cases a systemmay not be able to mix certain types of effects and so will play one ofthe effects instead. The system may reply on a priority measure toidentify which haptic effect among a plurality of haptics is mostimportant to be output. For example, if multiple haptic effects thathave similar frequencies or strengths that can be provided to theobserver, the priority system can determine which of the haptic effectsis the most important. The haptic effect may also be adjusted end pointwhere the effect is to be output. For instance, an effect may be stereoor mono or bidirectional or unidirectional depending on the capabilitiesof the haptic output device.

Haptic effects can be mixed together, based on the input parameters to anumber of emitters. A single haptic effect can be synthesized that issubstantially experienced by observers as equivalent to a concurrence ofthe multiple haptic sensations intended by the designers of theemitters. Haptic effects can be warped based on a medium that the hapticeffect is traveling through.

In some embodiments, the system is able to determine a type of hapticeffect based on a type of haptic output device. For example, if theuser's device is not capable of controlling a frequency of a hapticeffect, then the frequency of the haptic effect may not be changed. Asanother example, a user may be using a device that includes an advancedfriction display. The user can touch the display, and a haptic effectthat indicates a particular friction can be output. When another userobserves the same environment through their device, including the firstuser's finger moving through the environment, the second user's devicecan output a haptic effect that includes high-definition vibrationsrather than a change in friction.

In one embodiment, the haptic effects that are output may be based onproperties of a real world object. For instance, the haptic effect couldbe based on the density of an object. Other such properties mightinclude the shape, size, or certain user-defined properties.

Embodiments of the invention might comprise various types ofarchitectures for creation of objects and other aspects of theenvironment, modifications to objects and the environment, and creationand modifications to the haptic effects experienced by users of thesystem. For example, FIG. 1 shows a mobile device with a touch screen.However, in other embodiments, the device may not include a touchscreen. In some embodiments, a mobile device includes a sensor, such asa camera, that can be used for free-space gesture detection. In otherembodiments, the user may utilize a ring, watch, wristband, smartclothing, headset, or other wearable to interact with the system and theenvironment. In some embodiments, actuators may be embedded in any ofthose devices. In addition, a variety of haptic output device may beutilized including ultrasonic, laser, and jet projecting devices.

FIG. 3 is depicts an artist's rendering within a virtual or augmentedreality environment in one embodiment. In the rendering shown, theartist is able to use at least two brush styles for drawing or painting.The first brush is an oil-like paint brush that responds to sizecontrols and color customization. Such a brush allows the artist tocreate a landscape around in which to draw a vehicle route. The secondbrush style is a road or route. In the embodiment shown, the secondbrush allows the artist to create a route for spawned game object (car),each with different haptic properties.

The resulting environment may allow users to view the environment frommultiple perspectives, including as a content creator (e.g., artist),observer, or participant, for example. In one such embodiment, a fistuser uses a HMD to create content, while a second user uses a mobile paddevice to observe the environment. The content creator is able to feelhaptics from the act of mixing paint, painting, and putting down roads.The second user can use a mobile application window into the virtual oraugmented environment to observe or to make changes, such as spawningand controlling a car.

FIGS. 4 through 15 illustrate a process for mixing and applying colorsto an environment in one embodiment. FIG. 4 illustrates the results ofapplying color to a blank canvas. FIG. 5 illustrates a user interfacefor selecting colors in a virtual or augmented reality environment. Themixing of colors is performed in the illustrated embodiment using an inkwell metaphor. In such an embodiment, a user is able to select a colorfrom one of the swatches by placing their tracked finger into theswatch's “AR leaf.” The swatch animates to confirm that a selection hasbeen detected. In one embodiment, when the user hold down a brush“button” (normally used to draw) to activate an ink-dropper likeinteraction, the finger will “fill up” the longer the button is heldwhile the user holds their finger in the color swatch position (FIG. 6).The user is able to bring multiple colors into the finger without mixing(FIG. 7). Once the finger “contains” the color(s), the user can tap orapply pressure to the tablet surface to expel the color from the finger(FIG. 8). The color is then rendered on the tablet surface (FIGS. 9 and10).

In the embodiment shown, the user can use the finger to mix paint on thepalette (FIGS. 11 and 12). The user can also use a paint sponge toremove paint from the palette (FIG. 13). In one embodiment, the user canplace paint on the side of the palette to be used later (FIG. 14). Inone such embodiment, multiple colors placed on the palette may becombined using a swipe gestures (FIG. 15).

FIG. 16 is an illustration of a user interface for varying a tool in oneembodiment.

In the embodiment shown, the user is able to change the size of thebrush using gestures. For instance, to change the size of the brush, theuser swipes up or down on the left edge of the palette. Various gesturesmight be supported. For instance, the gesture may be curved. The size isbased on the length of the movement (i.e., the delta between thebeginning and end of the gesture); it is not mapped to specificpositions on the tablet of in space. In one embodiment, the size may bepreviewed over the virtual hand holding the brush tool.

FIG. 17 illustrates a palette for creating paths, such as roads, in oneembodiment. In the embodiment shown, a content creator, for instance, anartist wearing an HMD can “paint” various roads in a gaming environment.Once the roads are painted, other users can spawn vehicles to drivearound the drawn roads, feeling the haptics based on the road texturethe vehicle goes over. The user that creates the car may also controlthe car, causing it to start, accelerate, brake, and stop. FIG. 17 showsroad types that might be used, such as asphalt, dirt and cobblestone.These types are not exclusive. In other embodiments, the road types mayinclude streets, ravel, wet roads, rails, and flight paths.

FIGS. 18-24 illustrate various states that may be utilized in someembodiments. In the embodiment shown in FIG. 18, the device is in STATE0, which corresponds to off. In the off state, the palette has not beenactivated, and the only control shown is the “on” button.

FIG. 19 illustrates the digital pad and palette shown in another state,STATE 1, which is the paint state. In one such an embodiment, paintapplied to the tablet is saved when going to other states, so that itwill return upon return to painting state. The paint state allows a userto make changes to the environment, such as painting scenery.

FIG. 20 illustrates an embodiment that includes a STATE 2, which is astate for creating roads. In the embodiment shown, paint does notpersist in this state in order to provide enough room for roadselection. The purpose of roads in the embodiment shown is to provide aroute for mobile users to control a vehicle. Road “swatches” mayalternatively be selected with same leaf interaction as paint. The roadwidth may vary. For instance, the road may be broad enough toaccommodate two cars side-by-side.

However, in the embodiment shown, the brush size control may bedisabled. In one embodiment, a spawned car would attach to the “top” ofthe brush, so the “bottom” would not be textured. Also, in such anembodiment, as the brush is used to paint the road, one end of the roadwill connect to the end of another road or the other end of the sameroad to create a contiguous road. For instance, the ends of brushstrokes may snap to one another. In another embodiment, the two ends maybe automatically connected by a straight line. FIG. 20 illustrates thevarious road types displayed on a pad.

In some embodiments, the mobile application allows the user to spawn anobject to travel along the drawn roads. Then, based on the movement, thesurface, properties of the vehicle, and other properties of theenvironment, haptics are output. In one such embodiment, the spawnedvehicle begins at a set location on the road. For instance, the car maystart where the user creating the road begins the road.

FIGS. 21 and 22 illustrate an embodiment that includes a STATE 3, whichincludes a user interface for performing a state change. In theembodiment shown, a tertiary user interface (“UP”) provides that a usertilts a tablet 90 degrees so that it is perpendicular to the ground. Theuser is then presented with a variety of labels, such as paint, roads,and options. Each of these labels may include sub labels, for example,for options, the user may be presented with the following: save,auto-naming, clear drawing space, load, and exit. In variousembodiments, the user may interact in different ways. For instance, theuser may place a drawing finger in a state and click a button. Inanother embodiment, a user may hold a thumb button, traverse tabletposition in Y axis (so within one of the layers) and the release thebutton.

FIG. 22 illustrates another embodiment for performing a state change. Inthe embodiment shown, the user presses and holds a thumb button on apalette to freeze the various state boxes in space. Then the user movesthe tablet into a particular state box and releases to the button toselect a particular state. The states could include layers, such asdifferent colors or road types.

FIG. 23 illustrates an embodiment that includes a STATE 4, whichincludes a user interface for performing a quick action. The user firsttilts the tablet towards the user. In response, the swatches animate tobehind the tablet, and the paint melts away. Options that might beprovided by such an interface could include, undo, redo, eraser, clearpalate, and save.

FIG. 24 illustrates an embodiment that includes a STATE 5, whichincludes a user interface for loading and previewing saved files. Withsuch a user interface, the user can move the tablet vertically throughvarious layers to load particular files. In some embodiments, thedrawings can be previewed before loading. Once “load” is selected fromthe options panel, the user can return the tablet to normal operation.

FIGS. 25-32 illustrate various embodiments for interacting with avirtual environment with a “magic window” application. The magic windowapplication may be used to view a virtual or augmented realityenvironment. The application may be utilized by more than one usersimultaneously so that they can interact with other users in theenvironment. In some embodiments, the Magic Window allows viewers toview an artist's actions in the virtual environment as it happens. Inother environments, the viewer sees a collection of activities afterthey have happened in an asynchronous manner.

Embodiments of the Magic Window application provide haptic feedbackrelated to aspects of the environment. In some embodiments, such aspectsinclude other users' interactions with the virtual environment,including, for example, actions by the content creator. The Magic Windowapplication may also provide additional functionality, such as allowinga viewer to control aspects of the environment including controlling acar on a road as described above.

FIG. 25 illustrates a main menu display in one embodiment of a MagicWindow application. The main menu shown provides a menu, which allowsusers to find an artist to view, load a sketch (not MVP), and loadinformation about the application. The “Find Artist” option mayautomatically connect to a particular content creator or set of contentcreators or allow a viewer to search.

FIG. 26 illustrates an interface for grounding a view of a virtualenvironment. In the embodiment shown, the user initiates the applicationby, for example, connecting to a host. Then the user is prompted to scana visual marker (e.g., a QR code) to calibrate the viewing space andthereby grounding the view.

FIG. 27 illustrates the interface once the view is calibrated. Theviewer is able to see the content created by the content creator as wellas any changes occurring in real time. In the embodiment shown, theartist is visualized using a palette. The top left of the screenincludes an arrow for returning to the menu, such as the one illustratedin FIG. 25. And the top right includes icons illustrating the mode inwhich the interface is executing.

FIG. 28 is another embodiment of an interface displaying a rendering ofan environment that includes a road and spawned car. A car button isdisplayed at the top right of the screen and allows the user to enableor disable a “car” mode. The car mode allows the user to control theacceleration of a car on roads. The interface may also allow the user toview the environment from inside the car. Such an interface may allowmultiple users to concurrently spawn and control cars or other vehicles.

FIGS. 29 and 30 illustrate yet another embodiment of an interfacedisplaying rendering of an environment that includes a road and aspawned car. In the embodiment shown in FIG. 29, the viewer is viewingthe environment as if traveling in the car. As in the embodiment shownin FIG. 28, the user is able to control the car. This embodiment alsoprovides a representation of an artist, who is visualized as a palette,pen, and head on the upper left of the display. In FIG. 30, multipleusers have spawned cars, which are approaching one another on the road.

FIG. 31 is another embodiment and similar to the embodiment shown inFIG. 27. In the embodiment shown in FIG. 31, the application allows amaximum of two cars to be spawned on the road. If a user attempts tospawn another car, the application responds with an error message,informing the user that all slots are currently full.

FIG. 32 is an embodiment illustrating a more complex road and trafficimplementation. Such an embodiment may be useful in a multi-player racegame. Such an embodiment may comprise enhanced road physics, the abilityto fall off a map, the ability to crash into other cars, and multipleopponents. Other features that might be implemented include thecapability to boost the power of the car and to allow the car to driftor perform other activities based on particular physics model. In someembodiments, multiple different races may occur on the same virtualroad. In further embodiments, users may make changes to the road duringgame play, such as by making the road slick. Various other features ofconventional racing games may also be incorporated into embodiments ofthis invention. In some embodiments, a low-poly simplified style may bedesired for roads and cars, depending on how it looks with brush strokeenvironment.

FIGS. 33 and 34 illustrate embodiments of a content-creator applicationuser interface. Similar to the embodiments shown in FIG. 25-32, theembodiment shown in FIG. 33 allows a content creator a window into thevirtual environment. In the embodiment shown in FIG. 33, the contentcreator can tap the palette icon to enable or disable palette view. Thisallows the user to see a larger visualization of the artist's tabletconfiguration. FIG. 34 illustrates an interface for accessing sketches.In the embodiment shown, to open a sketch, the user enters the sketchselection dialog from the main menu and then taps on a preview of asketch to open.

In various embodiment, it is important to provide a compelling userinterface. For example, in some embodiments, the position of the paletteis tracked to the level of detail that an AR menu overlay can be appliedto its surface. In such an embodiment, the menu is animated with a highlevel of detail and polish. Objects do not appear and disappear, rather,they row, move, morph, and transition. In further embodiments, the menuincorporates the “AR Leaf” concept, whereby some menu items appear inthe space around the physical peripheral, and is interactive.

The menu may provide a variety of capabilities. For example, in oneembodiment, the menu provides the ability to select brush type. Inanother embodiment, interaction design cues are taken from physicalpaint mixing as opposed to desktop computing. Each of these capabilitiesmay be combined with compelling haptic feedback that is synchronized tothe menu's animations.

FIG. 35 illustrates a tablet that can be used in conjunction withvarious embodiments. The embodiment shown is a pressure sensitivewireless tablet. The tablet has a 31 mm casing for internals on theside, which will likely result in a user holding the tablet in aportrait orientation. The tablet may include an antenna at the top

The embodiments described herein provide numerous potential interactionsfor a user. For example, an embodiment may include an “off” state tosupport a suggested hand position to hold tablet, selecting color fromswatch “leaf,” placing an amount of selected color on the tablet, andrepeating process to mix and create/save new colors. Embodiments mayalso include state changes, gesture recognition, tool selection (e.g.,brushstroke), file operations, and other operations.

FIGS. 36-38 illustrate an embodiment in which the orientation in whichthe user holds the tablet controls the functionality available to theuser. In an embodiment that utilizes six degrees of freedom and threedimensionality around the palette, menus may be contextualized based onthe orientation that the tablet is held by the user. For example, suchan embodiment may incorporate three types of interactions: (1) a primaryinteraction: paint and brush interactions (normal orientation), (2) asecondary interaction: brush selection (angling down/looking over thetop), and (3) a tertiary interactions: save, clear, exit (angling tabletup to the left)

FIG. 36 illustrates an embodiment of the primary interaction, paint andbush sizing. FIG. 37 illustrates an embodiment allowing a user a secondorientation that presents the user with quick action options. Forexample, when a user tilts the tablet towards the user, swatches animateto behind the tablet, and paints melt away. In such an orientation,paints and materials do not persist. This orientation may provide thefollowing options: Undo, Redo, Eraser, Clear palette, and Save.

FIG. 38 illustrates yet a third orientation that allows a state change.This tertiary UI requires tilting tablet 90 degrees so that it isperpendicular to the ground. The user is then presented with severallabeled “layers”: Paint, Brushes, Materials, and Options. Further, theinterface may provide multiple options, such as: Save, Leaning towardsautomatic naming scheme to avoid text input need, Clear drawing space,Load, and Exit application (if needed). Various interaction may besupported in this orientation, including: place drawing finger in stateand click button, hold thumb button, traverse tablet position in Y axis(so within one of the layers) and release button.

FIG. 39 is an illustration of how a tablet may be laid out for use in avirtual or augmented reality environment in one embodiment. In theembodiment shown, the “AR Leaf” is at least 2″×2″. And the on-tablettouch interaction should be no smaller than 1″×1″.

FIG. 40 is an illustration of a primitive object used to generate morecomplex objects in a virtual or augmented environment in one embodiment.In the embodiment shown, a primitive 3D object—a cone—is presented tothe user. The user uses the cone to make a tree shape and thenrepositions the tree within the virtual or augmented environment.Various types of primitive objects may be presented in embodiments.

FIG. 41 is a flowchart illustrating a method 400 according to oneembodiment. In the embodiment shown, the system first determines theposition of an object in a virtual or augmented environment 402. Thesystem then determines a viewpoint of at least one observer with respectto the object 404. The system next determines a haptic effect to beoutput based at least in part on the position and the viewpoint 406. Andfinally, the outputs the haptic effect.

General Considerations

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process that is depicted as aflow diagram or block diagram. Although each may describe the operationsas a sequential process, many of the operations can be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Furthermore, examples of the methods may be implemented byhardware, software, firmware, middleware, microcode, hardwaredescription languages, or any combination thereof. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered. Accordingly, the above description doesnot bound the scope of the claims.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

Embodiments in accordance with aspects of the present subject matter canbe implemented in digital electronic circuitry, in computer hardware,firmware, software, or in combinations of the preceding. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises or has access to a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs including a sensor samplingroutine, selection routines, and other routines to perform the methodsdescribed above.

Such processors may comprise a microprocessor, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC),field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example tangible computer-readable media, that may storeinstructions that, when executed by the processor, can cause theprocessor to perform the steps described herein as carried out, orassisted, by a processor. Embodiments of computer-readable media maycomprise, but are not limited to, all electronic, optical, magnetic, orother storage devices capable of providing a processor, such as theprocessor in a web server, with computer-readable instructions. Otherexamples of media comprise, but are not limited to, a floppy disk,CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configuredprocessor, all optical media, all magnetic tape or other magnetic media,or any other medium from which a computer processor can read. Also,various other devices may include computer-readable media, such as arouter, private or public network, or other transmission device. Theprocessor, and the processing, described may be in one or morestructures, and may be dispersed through one or more structures. Theprocessor may comprise code for carrying out one or more of the methods(or parts of methods) described herein.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

1-7. (canceled)
 8. A method comprising: displaying a first user interface element in a virtual or augmented reality environment, the first user interface element comprising a plurality of layers along a first axis, wherein each of the plurality of layers is associated with a first property; receiving an input signal associated with a first movement of an object along the first axis; identifying the property associated with the layer corresponding to the position of the object.
 9. The method of claim 8, wherein the first property comprises an action and further comprising executing the action.
 10. The method of claim 9, wherein the action comprises a state change.
 11. The method of claim 8, further comprising displaying a second user interface element associated with an action.
 12. The method of claim 11, wherein the second user interface element comprises a button displayed in proximity to the first user interface element.
 13. The method of claim 11, wherein the second user interface element comprises a button for activating the first user interface element.
 14. The method of claim 8, wherein identifying the first property associated with the layer comprises determining that movement of the object along the first axis has stopped.
 15. The method of claim 8, further comprising identifying a second movement of the object in a second axis and executing an action in response to the movement in the second axis.
 16. The method of claim 15, wherein the second movement of the object in the second axis comprises a tilt.
 17. The method of claim 8, wherein the user interface further comprises: a plurality of layers along a second axis, wherein each of the plurality of layers is associated with a second property; receiving an input signal associated with a movement of the object along the second axis; identifying the second property associated with the layer corresponding to the position of the object.
 18. The method of claim 8, wherein the user interface element comprises a menu.
 19. The method of claim 8, wherein the object comprises a user-controlled indicator.
 20. The method of claim 8, further comprising outputting a haptic effect when the object approaches each of the plurality of layers along the first axis.
 21. The method of claim 8, further comprising outputting a haptic effect when the object is in proximity to each of the plurality of layers along the first axis.
 22. (canceled)
 23. A non-transitory computer-readable medium comprising processor-executable program code configured to cause a processor to: displaying a first user interface element in a virtual or augmented reality environment, the first user interface element comprising a plurality of layers along a first axis, wherein each of the plurality of layers is associated with a first property; receiving an input signal associated with a first movement of an object along the first axis; identifying the property associated with the layer corresponding to the position of the object.
 24. The non-transitory computer-readable medium of claim 23, wherein the first property comprises an action and further comprising executing the action.
 25. The non-transitory computer-readable medium of claim 24, wherein the action comprises a state change.
 26. The non-transitory computer-readable medium of claim 23, further comprising displaying a second user interface element associated with an action.
 27. The non-transitory computer-readable medium of claim 26, wherein the second user interface element comprises a button displayed in proximity to the first user interface element.
 28. The non-transitory computer-readable medium of claim 26, wherein the second user interface element comprises a button for activating the first user interface element.
 29. The non-transitory computer-readable medium of claim 23, wherein identifying the first property associated with the layer comprises determining that movement of the object along the first axis has stopped.
 30. The non-transitory computer-readable medium of claim 23, further comprising identifying a second movement of the object in a second axis and executing an action in response to the movement in the second axis.
 31. The non-transitory computer-readable medium of claim 23, wherein the user interface further comprises: a plurality of layers along a second axis, wherein each of the plurality of layers is associated with a second property; receiving an input signal associated with a movement of the object along the second axis; identifying the second property associated with the layer corresponding to the position of the object. 